Control of motion of an on-screen cursor, calibration of the motion control, and object selection using an on-screen cursor, in particular, passive remote control of on-screen cursor motion, including absolute position cursor calibration and on-screen object selection.
Input devices for cursor motion control and object selection on a computer screen are well known in the art. U.S. Pat. No. 4,886,941, entitled CIRCUIT FOR INTERFACING MOUSE INPUT DEVICE TO COMPUTER SYSTEM, issued to Davis et al on Dec. 12, 1989, the complete disclosure of which is incorporated herein by reference, for example, discloses a method for changing information from a motion sensor into quadrature signals for interfacing a mouse input device to a computer system via a channel designed to accept potentiometer-derived inputs. The inputs are typically transmitted via a hardwired electrical connection or a radio link, as disclosed in U.S. Pat. No. 5,854,621, entitled WIRELESS MOUSE, issued to Junod, et al on Dec. 29, 1998, the complete disclosure of which is incorporated herein by reference. The inputs control X-Y position signals for locating a cursor on the computer""s display screen, for example, to interface with xe2x80x9cmouse-iconxe2x80x9d user graphics, whereby the user moves the cursor to on-screen locations associated with user programs and commands. The xe2x80x9cmousexe2x80x9d is a form of hand-manipulated motion sensor very commonly used for cursor control. Other hand-manipulated motion sensors generally well known in the art include the xe2x80x9ctrack ballxe2x80x9d mouse and the xe2x80x9cjoy stick,xe2x80x9d examples of which are found on many laptop computers. Also well known is the requirement of hand manipulation of the mouse, track ball and joy stick to generate the motion signals that control the cursor location.
Each of these hand-manipulated motion sensors require that the user""s hand disengage from the keyboard, find the mouse, track ball or joystick, move the cursor, select the object, then reengage the keyboard. This sequence requires time and effort, especially in applications where both the keyboard and pointing are used extensively. The standard mouse also requires a significant amount flat, horizontal space to operate, and the moving parts are sensitive to dust and dirt, which interferes with precise control of the cursor.
One alternative hand-manipulated motion sensor is the xe2x80x9ctouchpad,xe2x80x9d also found on many laptop computers. As disclosed by U.S. Pat. No. 5,327,161, entitled SYSTEM AND METHOD FOR EMULATING A MOUSE INPUT, DEVICE WITH A TOUCHPAD INPUT DEVICE, issued to Logan, et al on Jul. 5, 1994, the complete disclosure of which is incorporated herein by reference, the touchpad is a mouse-emulating input device having a drag switch in which the direction of movement of touch across the touchpad surface causes a display cursor to move in the same relative direction as the movement of touch across the surface. While the touchpad overcomes some of the limitations of the standard mouse, the touchpad, like the small joysticks found on laptop computers, do not provide the precise cursor control or ease of movement normally desired by users. Furthermore, each of these cursor control methods is counter-intuitive: the body motion used to direct the device, and through it the cursor, is one not normally used to xe2x80x9cpointxe2x80x9d at an object. Rather, the user must move in a horizontal plane of motion that is rotated 90 degrees to the typical vertical plane of the display screen. This plane rotation requires a learning curve for the user. The typical user, for example, is not able to write words in a drawing program with a hand-manipulated device as well as by using a pen or pencil on paper.
Another problem with conventional hand-manipulated motion sensors is the manner of cursor positioning relative to the computer screen. According to this relative cursor positioning, physical motion of the mouse, or movement of touch across the touchpad surface, causes a responsive software application to move the cursor on the display screen in a direction consistent with the mouse or touch motion. When the mouse runs off of its friction pad or into an obstacle on the work surface or to the extent of the operator""s reach, the relative position of the mouse to the cursor must be reset, or xe2x80x9cre-centered,xe2x80x9d by lifting and repositioning the mouse on the friction pad. In a touchpad, when the operator""s digit moving across the touchpad reaches the limit of the touchpad""s surface, the relative position of the operator""s digit to the cursor must be similarly re-centered by lifting and repositioning the digit on the touchpad. In a mouse, this re-centering operation requires additional operator manipulation, while in a touchpad, re-centering increases the difficulty of the operator""s task in controlling the cursor.
Alternatively to conventional hand-manipulated motion sensors, optical cursor control systems are known that simplify manipulation of the cursor. In particular, eye-tracking devices, or xe2x80x9coculometers,xe2x80x9d are known that provide cursor manipulation by tracking eye movement. U.S. Pat. No. 4,109,145, entitled APPARATUS BEING CONTROLLED BY MOVEMENT OF THE EYE, issued to Graf on Aug. 22, 1978; U.S. Pat. No. 3,986,030, entitled EYE-MOTION OPERABLE KEYBOARD-ACCESSORY, issued to Teltscher on Oct. 12, 1976; and U.S. Pat. No. 5,325,133, entitled DEVICE FOR MEASURING A RETINA REFLECTED LIGHT AMOUNT AND A GAZE DETECTING apparatus using the same, issued to Adachi on Jun. 28, 1994, the complete disclosures of which are incorporated herein by reference, each disclose oculometers using light reflected from the user""s eye to produce signals representative of viewing direction. The eye tracker trains a camera on an eye to measure its movement, thereby detecting the direction an eye is gazing. Each of the above referenced and other known oculometers are limited by the amount of reflected light from the user""s eye.
As described in U.S. Pat. No. 4,891,630, entitled COMPUTER VISION SYSTEM WITH IMPROVED OBJECT ORIENTATION TECHNIQUE, issued to Friedman, et al on Jan. 2, 1990, the complete disclosure of which is incorporated herein by reference, two major drawbacks to the oculometer or eye tracking systems of the prior art are the field of view sensitivity and depth of field sensitivity. Field of view sensitivity requires the user to maintain a relatively constant position relative to the system""s optical axis, i.e., in line with the sensors, so that the user""s eye can be viewed. Depth of field sensitivity requires the user to maintain a constant position along the optical axis, i.e., a constant distance from the sensors, to remain in focus. Typically, the eye tracker uses the position of the corneal reflection relative to the center of the pupil to determine eye gaze location. However, because the pupil is very dark, lenses with a low xe2x80x9cfxe2x80x9d number have been used resulting in an eye tracker system with a small depth of field. Therefore, the user must keep the eye constantly in an unchanging position for the eye to remain in focus so that the eye tracking equipment can function effectively. Maximizing the depth of the field of view allows the user to move to various positions along the optical axis. One technique for improving the depth of the field is to use an automatic focusing lens, for example, a lens that will mechanically move in order to focus the image of objects at various depths along the optical axis. Alternatively, a fixed focus lens system is used to focus on an image of the user""s eye at only one depth. The depth of field inherent in the fixed focus system is maximized by increasing the brightness of the light reflected off the user""s eye.
U.S. Pat. No. 4,891,630 discloses a body pointing system that is combined with the eye tracker. Accordingly, U.S. Pat. No. 4,891,630 discloses a non-reflective, planar xe2x80x9corientation and position patchxe2x80x9d having at least three coplanar and non-collinear reflective spots A, B, and C, and a curved reflector D. The motion control device also includes a solid-state light-sensitive image sensor capable of generating a two-dimensional digital representation of the image stored in an array of memory cells arranged in multiple rows and columns. The cells each have an associated row and column address and the information at each cell is available for reading. Light from a source fixed relative to the sensor is directed onto the orientation and position patch, which is attached to an object, such as a human head or finger or an inanimate object. A lens fixed relative to the sensor defines an optical axis extending through the center of the lens and intersecting the image sensor near its center. The lens focuses a reflected light image of the object, including an image of the orientation and position patch, upon the image sensor.
The system further requires a computer having a processor and main memory and the capability for reading the row and column addresses of memory cells corresponding to the images Axe2x80x2, Bxe2x80x2, and Cxe2x80x2 of reflective spots A, B, and C and an image DLS of the light reflected from the curved reflector D, thereby developing a binary image of each image. A task stored in the computer memory detects and quantifies changes in orientation and position of that surface of the object upon which the orientation and position patch is secured. The task detects changes in the position along the optical axis and in directions perpendicular thereto by computing changes in the row and column addresses of images Axe2x80x2, Bxe2x80x2, and Cxe2x80x2. The task also detects rotational changes about axes perpendicular to the optical axis by computing differences in the row and column addresses of the image DLS and the images Axe2x80x2, Bxe2x80x2, and/or Cxe2x80x2.
An eye tracking device is formed when the orientation and position patch is fixed on a user""s face in relative proximity to the user""s eye. A reflected light image of the eye formed by the lens upon the image sensor is analyzed to determine the direction of gaze by detecting the position of the corneal reflection relative to the position of the head. The position of the corneal reflection CR is correlated to the position being gazed upon. Head orientation is computed from a view of the reflective spots on the patch and the corneal reflection CR. The image processing hardware and software extracts the reflections from a single camera image and computes the x and y positions of the center of each reflection. Software operated by the microprocessor reads the locations of edges in the binary images from memory, performs a standard, binary image region connectivity analysis, and computes x and y centers for each image. Regions representing the patch reflections and the corneal reflection are selected and labeled. The x and y differences between the centers of regions that are necessary to compute the user""s head orientation and point of gaze are then computed.
A two-position calibration sequence in necessary when the eye tracker is engaged. The user, wearing the reflective patch beneath the eye, sits in front of the camera lens. First, the user looks at a top left selection position and second, at a bottom right selection position. After this calibration, a position on the front panel of the device are selected by directing the user""s gaze at it. The computer uses the position of the corneal reflection relative to the reflective patch to compute head position and orientation changes. However, if the user moves outside the field of the camera lens, no means is provided for retaining the calibration information so that the calibration sequence must be repeated in whole. Furthermore, the above described need to focus light on the user""s eye and obtain the reflected image severely limit the useful field of view and depth of field of the oculometer eye tracker systems.
Thus, cursor control systems have been developed that use a photo sensor to optically track the image of small reflective or emissive targets fixed relative to the user, usually attached to the head. Position information derived from the emitted or reflected light images controls the movement of a cursor on a computer or other system""s display screen, thereby performing cursor motion control functions normally associated with a hand or eye-manipulated device. Attached to the user""s head, the target device becomes a xe2x80x9chead tracker,xe2x80x9d as described in U.S. Pat. No. 4,891,630. When the head moves, a responsive software command moves the cursor from its last position in a direction consistent with the head motion. Thus, like the mouse and other hand-manipulated cursor control devices, the head tracker also requires body motion to direct the device, and through it the cursor, that is not normally use to xe2x80x9cpointxe2x80x9d at an object. Unlike the oculometer xe2x80x9ceye trackerxe2x80x9d systems, rather than just the eye, the user must move the entire head to point. This head motion requires a learning curve for the user. The typical user, for example, is not able to write words in a drawing program with a hand-manipulated device as well as by using a pen or pencil on paper.
While relative cursor motion by sensing of a remote target is thus provided in the prior art, there remains the need to interact with the software application by making a selection and indicating this choice to the operating software application. Motion of the cursor on the display screen does not normally accomplish this task. In other words, mere cursor motion control does not provide a method of selecting an object. Rather, additional circuits and software must be present to perform the selection process. Typically, the selection process is performed using a mouse by the user depressing or xe2x80x9cclickingxe2x80x9d the mouse buttons to activate this additional circuitry which is integral with the cursor motion control in the mouse. The additional circuitry in turn transmits control command signals to the control software which is typically resident on the computer. Prior art methods for performing this selection or xe2x80x9cclickingxe2x80x9d task include time based selection in which a click is assumed if the cursor is stationary, or dwells, over an object for a predetermined period of time. For example, U.S. Pat. No. 4,891,630 discloses an oculometer or eye tracker system wherein selection is made when the cursor dwells over the selection. Alternatively, a separate cable controlled cursor is used to transmit the one or more necessary commands.
Selection or activation of a software application often requires additional clicks performed in rapid sequence. Many current mouse designs include additional integral circuitry for xe2x80x9cdouble clickingxe2x80x9d whereby the rapid sequence of additional clicks is provided in a single motion by the user depressing or xe2x80x9cclickingxe2x80x9d a second mouse button to activate this additional circuitry. Additionally, separate roller switch-controlled circuitry provides a xe2x80x9cscrollingxe2x80x9d function. One such example is the MouseMan(copyright) model M-CW47 available from Logitech of Fremont, Calif.
The present invention overcomes the cursor control limitations of the prior art by providing an optical device for tracking a remote object. The device of the invention includes an optical sensor formed of a number of photosensitive pixels arraigned in a two-dimensional array of rows and columns; an imaging lens arranged to image onto the optical sensor a cursor control object positioned within the field of view and the depth of field of the imaging lens; and an interface circuit coupled to the optical sensor for a) determining position information describing position of the cursor control object image relative to the two-dimensional array of pixels, and b) producing cursor control signals suitable for controlling the position of a cursor on a computer-controlled display as a function of the control object image position information.
According to one aspect of the invention, the optical tracking device also includes an infrared radiation source essentially collocated with the imaging lens and directing infrared radiation into at least a portion of the field of view of the imaging lens. Thus, the cursor control object is illuminated by the infrared radiation, and the imaging lens images reflected infrared radiation from the illuminated cursor control object onto the optical sensor.
According to one aspect of the invention, the cursor control object is a digit of the user""s hand. The user""s digit either relies upon the natural reflectivity of the user""s skin or is augmented with an attached reflecting device. In a vertical finger pointing mode, the user moves the digit, with or without augmentation, within the field of view of the imaging lens. The motion of the digit is translated into cursor motion on a display screen when the radiation reflected from the digit is imaged onto the optical sensor and converted into an electrical video signal by the interface circuit, which is interpreted by a conventional cursor motion control function implemented in a software program and operated on a processor interfaced to the display screen. The control motion practiced by the user is a natural pointing motion, preferably using the index finger of either hand. Furthermore, when practiced in combination with a standard computer keyboard, the cursor control motion is accomplished without disengaging the hands from the keyboard.
According to another aspect of the invention, when practiced in a horizontal finger pointing mode, the radiation sources and imaging lens are located on a surface of the keyboard, and the digit control motion is to that used with a standard touchpad. But the cursor control is accomplished without touching any surface.
According to other aspects of the invention, the interface circuit includes one or more filters for implementing signal thresholds to distinguish object images that have an intensity above a predetermined threshold intensity. Also, the circuit includes circuitry for each of determining connectivity between the various pixels determined to have an intensity above the predetermined threshold intensity, determining connectivity between the pixels, determining a centroid of each object image, and generating output signals representative of the position of the centroid of each object image relative to the array of photosensitive pixels of the optical sensor.
According to other aspects of the invention, the invention provides a signal processing function implemented in software for operation on a processor, the signal processing function is electrically coupled to receive the output signals representative of the position of the centroid of each object image relative to the optical sensor and determines one control object image among one or more of the object images. The processor also operates one or more elimination algorithms in determining one control object image among multiple object images. Furthermore, the processor operates a function implemented in software for determining one control object image among the object images as a function of the physical arrangement of the user""s digit or digits relative to the field of view of the imaging lens. According to still other aspects of the invention, the processor also operates a function implemented in software for identifying one of the object images appearing closest to a predetermined edge of the optical sensor in order to determine the control object image.
According to another aspect of the invention, the invention includes a cursor motion control function implemented in software for operation on a processor, the cursor motion control function responding to an output signal of the signal processing function representative of the position of the centroid of the control object image relative to the array of photosensitive pixels of the optical sensor for controlling a position of a cursor relative to a display screen.
According to still other aspects of the invention, the invention provides a clicking function implemented in software for operation on a processor, the clicking function of the invention generates a signal imitating a conventional mouse click signal in response to a change in the control object image. According to various aspects of the invention, the change in the control object image is any one or a combination of: a) motion of the control object image in predetermined pattern; and b) a change in the quantity of images having an intensity above the predetermined threshold intensity, including i) a reduction in the intensity of the control object image to less than the predetermined threshold intensity during a predetermined term followed by a restoration of the intensity above the threshold in essentially the same position relative to the optical sensor, i.e. a momentary flashing xe2x80x9coffxe2x80x9d of the image; and ii) a momentary appearance of one or more additional object images having an intensity above the threshold in essentially a predetermined position relative to the control object image.
According to yet another aspect of the invention, the cursor calibration limitations of the prior art are overcome by the present invention by providing a calibration sequence function implemented in software for operation on a processor. The calibration sequence function determines a current position of the control object relative to the array of pixels of the optical sensor, and maps the pixel containing the centroid of the control object image onto a predetermined pixel of the display screen. Preferably, the calibration sequence function maps the centroid of the control object image to a pixel at the approximate center of the display screen.
According to a further aspect of the invention, the calibration sequence function responds to an input signal, for example, a mouse click signal, indicating an instant at which to determine the current position of the control object relative to the optical sensor, whereby the user is able to select the desired physical position of the cursor control object in space relative to the imaging lens.
According to another aspect of the invention, the calibration sequence function also defines a relationship between the amount of spatial movement of the control object relative to the optical sensor and the cursor motion on the display screen. The relationship is defined either by a default setting in the implementation software or a user selected setting.
According to other aspects of the invention, the cursor control object is a source of electromagnetic radiation positioned within the field of view of and remotely from the imaging lens. Emitted radiation from the cursor control object is directed into the imaging lens and preferably forms an object image on the two-dimensional array of photosensitive pixels forming the optical sensor. The object image having an intensity above the predetermined threshold intensity. A centering function implemented in software for operation on a processor determines a current position of the object image relative to the array of pixels of the optical sensor and maps the pixel containing a centroid of the object image to a predetermined pixel of the computer-controlled display screen. Preferably, the centering function maps the pixel containing a centroid of the object image to a pixel at the approximate center of the display screen. According to one aspect of the invention, the invention further includes a cursor motion control function implemented in software for operation on a processor, the cursor motion control function responds to an output signal of the centering function representative of the position of the centroid of the object image relative to the array of photosensitive pixels of the optical sensor to control the cursor""s position relative to the display screen. According to another aspect of the invention, the invention further includes a clicking function implemented in software for operation on a processor, the clicking function generates a signal imitating a mouse click signal in response to a change in the object image. For example, the imitation mouse click signal is generated when a steady radiation emission is interrupted for a predetermined minimum term.